Feature Selection , Empirical Risk Minimization , and The Orthogonal Case
نویسنده
چکیده
Recall that: L(w) = 1 n E‖Xw − Y ‖ = 1 n E‖Xw − E[Y ]‖ + σ Define our “empirical loss” as: L̂(w) = 1 n ‖Xw − Y ‖ which has no expectation over Y . Note that for a fixed w E[L̂(w)] = L(w) e.g. the empirical loss is an unbiased estimate of the true loss. Suppose we knew the support size q. One algorithm is to simply find the estimator which minimizes the empirical loss and has support only on q coordinates. In particular, β̂q = inf support(w)≤q L̂(w)
منابع مشابه
Mental Arithmetic Task Recognition Using Effective Connectivity and Hierarchical Feature Selection From EEG Signals
Introduction: Mental arithmetic analysis based on Electroencephalogram (EEG) signal for monitoring the state of the user’s brain functioning can be helpful for understanding some psychological disorders such as attention deficit hyperactivity disorder, autism spectrum disorder, or dyscalculia where the difficulty in learning or understanding the arithmetic exists. Most mental arithmetic recogni...
متن کاملSublinear Models for Graphs
This contribution extends linear models for feature vectors to sublinear models for graphs and analyzes their properties. The results are (i) a geometric interpretation of sublinear classifiers, (ii) a generic learning rule based on the principle of empirical risk minimization, (iii) a convergence theorem for the margin perceptron in the sublinearly separable case, and (iv) the VC-dimension of ...
متن کاملSparse Support Vector Infinite Push
In this paper, we address the problem of embedded feature selection for ranking on top of the list problems. We pose this problem as a regularized empirical risk minimization with p-norm push loss function (p = ∞) and sparsity inducing regularizers. We leverage the issues related to this challenging optimization problem by considering an alternating direction method of multipliers algorithm whi...
متن کاملOptimal oracle inequalities for model selection
Abstract: Model selection is often performed by empirical risk minimization. The quality of selection in a given situation can be assessed by risk bounds, which require assumptions both on the margin and the tails of the losses used. Starting with examples from the 3 basic estimation problems, regression, classification and density estimation, we formulate risk bounds for empirical risk minimiz...
متن کاملMulti-Objective Programming in SVMs
We propose a general framework for support vector machines (SVM) based on the principle of multi-objective optimization. The learning of SVMs is formulated as a multiobjective program by setting two competing goals to minimize the empirical risk and minimize the model capacity. Distinct approaches to solving the MOP introduce various SVM formulations. The proposed framework enables a more effec...
متن کامل